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ABSTRACT 

Operation  Iraqi  Freedom  and  Operation  Enduring 
Freedom  have  identified  the  need  for  instructional  and 
training  solutions  that  develop  the  skills  of  Battalion  and 
Brigade  Commanders  in  formulating  situational 
understanding  in  order  to  successfully  lead  operations  in  a 
counterinsurgency  environment.  In  this  paper  we  describe 
the  UrbanSim  Learning  Package,  a  game-based 
instructional  software  suite  for  Commanders  and  their 
staffs  for  directing  and  coordinating  full-spectrum 
operations  where  the  stability  component  is  predominant. 
We  describe  a  formal  instructional  design  approach  to  the 
development  of  this  instructional  software,  which  consists 
of  a  component  that  introduces  key  concepts  in 
counterinsurgency  operations  and  a  component  that 
allows  students  to  develop  their  skills  in  a  simulated 
counterinsurgency  environment.  We  describe  how 
intelligent  automated  tutoring  is  used  to  provide  formative 
feedback  to  students  in  the  practice  environment,  and 
discuss  our  approach  to  student  performance  assessment. 

1.  INTRODUCTION 

In  the  book  Fiasco,  Thomas  Ricks  points  out  that 
many  of  the  tactics  initially  employed  by  U.S. 
commanders  in  2003  “led  away  from  the  strategic  goal  of 
winning  the  political  support  of  the  Iraqi  people”  (Ricks, 
2006).  Conventional  tactics  achieved  overwhelming 
successes  in  the  initial  stages  of  operations  in  Iraq. 
However,  it  quickly  became  apparent  that  the 
technological  advantages  of  the  U.S.  forces  were 
countered  by  the  unconventional  tactics  of  an  insurgency. 
The  U.S.  forces  were  going  to  need  to  “relearn  the 
principles  of  counterinsurgency  (COIN)  while  conducting 
operations  against  adaptive  insurgent  enemies.”  (FM-3- 
24,  2006).  This  need  led  to  the  development  of  the 
Counterinsurgency  Field  Manual,  FM  3-24/MCWP  3- 
33.5  to  better  understand  insurgent  operations  and  tactics 


to  counter  these  operations  through  a  combination  of 
offensive,  defensive,  and  stability  means.  Additionally, 
significant  updates  were  made  to  the  Operations  Field 
Manual  (FM  3-0),  which  now  dedicates  an  entire  portion 
to  a  discussion  of  the  tactics,  techniques,  and  procedures 
for  planning,  preparing,  and  conducting  full-spectrum 
operations  (FM  3-0,  2008). 

Today’s  Army  leaders  face  extremely  stressful  and 
demanding  missions  that  are,  in  many  cases,  not  covered 
by  standard  tactics  and  doctrine.  These  operations,  which 
combine  both  lethal  and  non-lethal  aspects  of  warfare,  are 
referred  to  by  Dr.  David  Kilcullen  as  “armed  social 
work,”  in  which  U.S.  and  Host  Nation  forces  attempt  to 
“redress  basic  social  and  political  problems  while  being 
shot  at”  (Kilcullen,  2006).  The  training  challenge  is  to 
develop  adaptable  leaders  who  function  effectively  in 
complex  environments  and  succeed  in  novel  situations 
unlike  any  they  may  have  experienced  in  the  past 
(ATDLP,  2003;  Wong,  2004). 

In  this  paper,  we  describe  the  UrbanSim  Learning 
Package,  a  game-based  instructional  software  suite  for 
battalion  and  brigade  commanders  and  their  staffs  for 
directing  and  coordinating  full-spectrum  operations  with  a 
“stability-focused”  component.  UrbanSim’ s  focus  is 
predominantly,  but  not  exclusively,  on  non-lethal 
operations  in  support  of  the  local  citizenry  and 
government  that  take  place  after  primary  offensive  and 
defensive  efforts  have  concluded.  We  discuss  the 
instructional  approach,  the  design  and  development  of  the 
student  experience,  mechanisms  for  measuring 
performance,  the  role  of  intelligent  tutoring  and 
assessment,  and  our  lessons  learned  throughout  the 
instructional  design  lifecycle.  Although  the  instructional 
and  system  designs  employed  for  UrbanSim  target  the 
battalion  and  brigade  command  level,  there  are 
generalities  within  the  approach  that  make  it  suitable  for  a 
variety  of  stability-focused  domains,  including 
interagency  and  joint  operations,  higher  echelon 
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command  (Corps  and  Division),  and  even  non-military 
operations  (non-government  organizations.  State 
Department). 

2.  INSTRUCTIONAL  DESIGN 

The  UrbanSim  Learning  Package  has  adopted  a 
formal  instructional  design  approach  to  guide  the 
development  of  the  underlying  experience.  This  design 
approach  has  proven  successful  in  the  development  of  the 
ELECT  BiLAT  training  system  (Hill  et  al.,  2006),  a 
precursor  to  UrbanSim  that  focuses  bilateral  negotiations 
with  cultural  awareness.  Building  on  the  lessons  learned 
from  ELECT  BiLAT  and  applying  the  principles  of 
Guided  Experiential  Learning  (Clark,  2004),  a  set  of 
processes  were  employed  that  would  determine  how  the 
UrbanSim  instructional  software  would  be  developed, 
tested,  deployed,  and  used  in  a  classroom  setting.  This 
section  describes  the  seven  steps  of  this  process,  including 
lessons  learned,  as  they  relate  to  UrbanSim. 

2.1  Step  1:  Identify  the  training  context 

The  first  step  in  the  design  approach  is  to  identify  and 
understand  the  context  in  which  the  instructional  software 
will  be  used.  For  UrbanSim,  the  target  training  audience 
is  students  in  the  Tactical  Commander  Development 
Program  at  the  School  for  Command  Preparation,  Ft. 
Leavenworth.  Instructors  highlighted  the  need  to  give 
future  battalion  and  brigade  commanders  attending  this 
course  the  opportunity  to  practice  directing  COIN 
operations.  While  sufficient  curricula  materials  already 
existed  for  traditional  symmetric,  force-on-force  style 
operations  (i.e.,  offense,  defense),  there  was  a  gap  in  less 
traditional  stability-focused  operations.  As  a  result,  it  was 
determined  that  UrbanSim  would  target  this  area  and 
include  the  capacity  to  provide  practice  in  directing 
operations  in  these  kinds  of  missions.  The  goal  identified 
for  the  UrbanSim  instructional  package  was  to  provide 
future  battalion  and  brigade  Commanders  with  an 
interactive  experience,  allowing  them  to  practice  full- 
spectrum  operations  where  stability  is  predominant. 

After  identifying  the  training  domain,  the  integration 
of  UrbanSim  into  the  current  program  of  instruction  was 
considered.  This  involved  determining  how  much  time 
could  be  devoted  to  the  topic  (both  in  class,  and  as 
assigned  work  out  of  class)  and  how  to  realign  the  current 
program  of  instruction  so  as  to  open  up  this  time. 

Next,  an  initial  set  of  learning  objectives  was 
identified  for  the  educational  package.  A  learning 
objective  is  a  “statement  of  what  students  will  be  able  to 
do  when  they  have  completed  the  instruction”  and  is 
articulated  as  quantifiable,  measurable  verbs  that  signify  a 
demonstrable  learning  outcome  (Arreola,  1998). 
Formulation  of  the  initial  learning  objectives  required 


several  discussion  sessions  with  instructors  and  subject 
matter  experts.  These  learning  objectives  focused  on  how 
to  develop  situational  understanding,  develop  and 
describe  the  Commander’s  visualization,  think  during 
decision-making  and  directing  actions,  and  self-assess 
progress  in  stability-focused  environments. 

2.2  Step  2:  Conduct  a  Cognitive  Task  Analysis 

Upon  formulation  of  the  initial  set  of  learning 
objectives,  a  Cognitive  Task  Analysis  (CTA)  was 
conducted.  A  CTA  is  a  process  that  employs  specific 
interview  and  observation  strategies  to  extract  both  the 
explicit  and  implicit  knowledge  that  experts  use  to 
perform  a  certain  task  (Schraagen  et  al.,  2000).  The  task 
associated  with  commanding  a  battalion  is  defined  as 
complex  because  it  requires  the  use  and  coordination  of 
both  controlled  (conscious)  and  automated  (procedural) 
knowledge  that  extends  over  a  long  period  of  time  (Van 
Merrienboer  et  al.,  2002).  The  results  from  a  CTA 
describe  the  detailed  skills  required  to  successfully 
complete  the  complex  task.  The  intent  is  to  organize  and 
give  meaning  to  observable  behavior  through  the 
“unpacking’  of  knowledge.  For  UrbanSim,  CTA 
interviews  were  conducted  with  experts  that  included 
battalion  Commanders  who  recently  returned  from  the 
Iraqi  theatre.  Through  these  interviews,  the  goals  of 
UrbanSim  were  further  refined  to  include  gaining  and 
maintaining  the  support  of  the  local  population  within  an 
area  of  operations  to  achieve  a  safe  environment  secured 
and  governed  by  the  Host  Nation.  The  CTA  also  multiple 
measures  are  used  to  gauge  success,  including  Measures 
Of  Effectiveness  (MOE)  and  Measures  Of  Performance 
(MOP).  The  sequence  of  procedures  used  to  accomplish 
this  task  were  to  1)  conduct  mission  analysis,  2)  conduct 
deliberate  reconnaissance,  3)  review  and  approve 
targeting-synchronization  matrix,  and  4)  assess  progress 
toward  achieving  overarching  outcomes. 

These  high-level  procedures  laid  the  framework  for 
the  package’s  final  set  of  terminal  learning  objectives  and 
provided  the  pedagogical  foundation  for  the  detailed 
instructional  design  in  Step  4  below.  The  final  set  of 
terminal  learning  objectives  were  as  follows: 

1.  The  Population  is  the  Center  of  Gravity:  the  student 
should  understand  how  to  create  a  safe,  secure,  and 
prosperous  population  that  is  the  cornerstone  to  self- 
sustainment  and  a  functioning  Host  Nation 
government. 

2.  Conducting  mission  analysis:  the  student  should 
employ  higher  headquarters  products  and  running 
staff  estimates  to  formulate  and  continuously  update 
a  complete  situational  understanding  and 
visualization  of  the  battlefield. 

3.  Conduct  mission  planning:  the  student  should 
develop  the  set  of  products  that  allows  them  to 


articulate  a  desired  end  state  and  courses  of  action  for 
a  COIN  operation. 

4.  Directing  action :  student’s  use  staff,  subordinate,  and 
higher  headquarters  updates  to  direct  action  in  a 
COIN  environment  to  a)  continuously  update  and 
refine  the  student’s  situational  understanding,  b) 
advance  towards  achieving  their  desired  end  state, 
and  c)  anticipate  2nd  and  3rd  order  effects. 

5.  Assessing  performance:  the  student  should  employ 
lines  of  effort  (LoEs),  MOEs,  and  MOPs  to  self- 
assess  mission  progress  in  a  COIN  operation. 

2.3  Step  3:  Formulate  the  initial  instructional  design 

In  this  design  approach,  the  task  objective,  procedures, 
and  specific  tasks  identified  by  the  CTA  (along  with  the 
terminal  learning  objectives)  are  used  to  develop  an 
instructional  design  that  operates  within  the  limitations  of 
the  training  context  (see  step  1).  For  UrbanSim,  these 
considerations  led  us  to  an  instructional  design  that 
consisted  of  two  separate,  but  intertwined,  components:  a 
Primer  and  Practice  Environment.  Though  the  CTA 
identified  specific  tasks  that  a  Commander  could  take  to 
be  successful  in  a  COIN  operation,  it  was  found  in  steps  1 
and  2  that  students  entering  the  Tactical  Commander 
Development  Program  at  the  School  for  Command 
Preparation  required  additional  practice  to  regain  the 
working  and  conceptual  knowledge  necessary  to  execute 
these  tasks  successfully.  Therefore,  a  Primer  was 
introduced  to  provide  a  student  with  the  core  conceptual 
and  associated  task  knowledge  that  would  allow  them  to 
be  successful  in  the  game-based  Practice  Environment, 
described  further  in  section  3. 

2.4  Step  4:  Performance  measurement 

For  an  instructional  application  such  as  this,  the 
primary  reason  to  measure  performance  is  to  gather  data 
that  can  be  used  to  give  the  student  feedback  and  to  steer 
their  future  thinking  in  the  right  direction.  There  are  two 
kinds  of  performance  measures  that  can  be  used  in  this 
manner:  outcome  measures  and  process  measures. 
Outcome  measures  are  derived  directly  from  the  state  of 
the  world.  One  type  of  outcome  measure  is  factual 
information  about  the  environment.  For  example,  how 
many  IEDs  were  detected  this  week  compared  to  last 
week?  The  other  type  includes  attitude  measures  from  the 
local  simulated  populace.  The  intelligent  agents  in 
UrbanSim  Practice  Environment  have  beliefs  and  goals; 
therefore  we  can  “poll”  the  simulated  population  to  assess 
their  attitudes  (e.g..  Do  you  support  attacks  on  the  host 
nation  military?).  Because  the  ultimate  goal  of  COIN 
operations  is  to  gain  the  population’s  support  for  the  host 
nation  government,  and  this  is  one  of  the  main  points  we 
want  our  students  to  appreciate,  it  is  important  to  be  able 
to  track  the  course  of  such  measures,  and  use  them  to 
provide  feedback  to  students. 


Process  measures  are  concerned  less  with  the  state  of 
the  world,  and  more  with  the  way  in  which  students  make 
decisions.  The  training  goal  is  to  shape  student  thinking  to 
follow  the  themes  (or  terminal  learning  objectives)  falling 
out  of  the  CTA  (section  2.2).  In  a  computer-based 
learning  environment,  however,  we  are  limited  to  the 
interactions  between  the  student  and  the  user  interface.  It 
is  therefore  important  to  design  the  user  interface  so  that 
observable  user  interactions  support  inferences  about 
underlying  cognitive  processes.  For  example,  if  a  student 
never  takes  an  action  to  advance  a  particular  LoE,  it  can 
be  inferred  that  advancement  of  this  LoE  does  not  figure 
prominently  in  their  decision-making.  To  infer  attention 
to  the  different  LoEs,  then,  the  interface  (and  game 
mechanics)  should  support  which  LoEs  the  student  is 
primarily  trying  to  advance  with  each  action  taken.  There 
are  several  challenges  involved  in  designing  these  types 
of  measures  into  the  training  system.  First  and  foremost  is 
determining  inference  rules  by  which  cognitive  processes 
(or  lack  of  them)  can  be  inferred  from  patterns  of  system 
interactions.  Another  challenge  is  designing  those 
measurement  opportunities  into  the  system  without  them 
becoming  so  intrusive  as  to  undermine  the  flow  of  the 
simulated  mission.  The  most  important  outcome  measure 
used  in  UrbanSim  is  how  much  time  the  student  spends  in 
each  of  the  various  user  interface  displays.  This  value  is 
tracked  during  the  course  of  a  scenario  to  determine 
whether  emphasis  is  placed  to  heavily  on  a  particular  type 
of  view  (such  as  the  map),  while  disregarding  the  other 
views  critical  to  the  mission  (network  display,  LoEs, 
MOEs,  running  staff  estimates,  etc). 

2.5  Step  5:  Identify  technologies  and  develop  a 

prototype 

Once  the  instructional  design  has  been  created  and 
performance  criteria  identified,  a  brief  but  thorough 
review  of  appropriate  technologies  should  be  conducted 
to  identify  candidates  that  best  meet  the  pedagogical 
needs  of  the  classroom.  Performing  this  step  before  an 
instructional  design  has  been  created  poses  significant 
risks  because  the  design  ends  up  being  constrained  by  the 
limitations  (or  features)  of  the  technology.  Immediately 
after  identification  of  candidate  technologies,  a  prototype 
should  be  rapidly  developed  and  delivered  to  the 
instructors  to  highlight  the  features  of  the  instructional 
design.  Rapid  prototypes  can  also  assist  with  managing 
expectations  by  providing  the  customer  with  a  preview  of 
the  courseware  and  its  functionality.  With  assistance  from 
the  University  of  Southern  California’s  Game  Innovation 
Lab,  a  Microsoft  Excel-based  application  was  created  that 
allowed  a  player  to  direct  action  with  subordinate  units  in 
a  COIN  operation  and  see  effects  on  the  world  state.  From 
here,  a  primitive  virtual  representation  of  this  Excel-based 
program  was  created  with  an  off-the-shelf  2D  game 
engine.  Although  this  early  prototype  differed 
significantly  from  the  final  version  of  the  game-based 


Practice  Environment,  it  did  provide  instructors  with  a 
concrete  idea  of  the  intended  learning  experience.  It  also 
supplied  the  instructional  design  team  with  input  from  the 
classroom  regarding  modifications  to  the  mechanics  and 
interactions  with  the  software. 

2.6  Step  6:  Production 

After  the  prototype  has  been  demonstrated  and  input 
received  from  the  classroom,  the  instructional  design 
formulated  in  Step  4  is  adjusted  and  finalized.  Production 
then  commences  with  development  of  the  underlying 
courseware,  always  keeping  the  classroom  abreast  of 
progress  and  adjustments  to  the  instructional  design.  This 
communication  proves  invaluable  as  the  classroom  is  able 
to  inform  the  design  team  of  potential  changes  in  the 
curriculum,  which  may  or  may  not  require  shifts  in  the 
design.  Upon  development  completion,  various  forms  of 
testing  takes  place,  ideally  within  the  classroom  when  the 
courseware  is  in  a  stable  state.  Finally,  initial  deployment 
to  the  classroom  occurs  after  testing  and  quality  assurance 
is  complete.  The  first  versions  of  the  two  UrbanSim 
instructional  components  (Primer  and  Practice 
Environment)  were  produced  at  the  USC  Institute  for 
Creative  Technologies  along  with  industry  partners,  and 
are  described  in  section  3. 

2.7  Step  7:  Courseware  assessment 

After  initial  deployment,  user  testing  and  assessment 
is  conducted  that  will  provide  suggestions  for  the 
refinement  of  the  user’s  experience  which  may,  or  may 
not,  include  refinement  of  the  instructional  design.  This  is 
a  critical  step  in  the  courseware  lifecycle  process  as  it 
ensures  the  courseware  does  not  become  stale  over  time, 
or  as  the  course  makes  changes  to  its  curriculum.  Though 
this  refinement  cannot  occur  in  perpetuity,  it  is  important 
to  keep  the  channels  of  communication  with  the 
classroom  open  long  after  the  courseware  is  delivered. 
Strategies  for  assessment  of  the  UrbanSim  Primer  and 
Practice  Environment  are  discussed  in  section  5. 

3.  URBANSIM  SYSTEM  DESIGN 

As  discussed,  UrbanSim  consists  of  two  separate,  but 
intertwined,  components.  The  Primer  introduces  the 
student  to  the  core  principles  and  associated  tasks  for 
conducting  stability-focused  operations  using  interactive 
video  clips,  stories,  part-task  practice  sessions,  and 
reading  materials.  The  Practice  Environment  allows  the 
student  to  conduct  such  an  entire  operation,  from  planning 
through  assessment.  Employing  the  principles  from 
Guided  Experiential  Learning,  these  two  components 
provide  the  student  with  the  core  concepts  and  task 
knowledge  necessary  to  command  units  within  the 
contemporary  operating  environment. 


3.1  The  UrbanSim  Primer 

The  UrbanSim  Primer  introduces  the  core  concept  and 
task  knowledge  required  for  a  future  battalion  commander 
to  successfully  command  in  a  COIN  environment.  It  takes 
the  form  of  an  interactive  tutorial  that  guides  a  student 
through  the  required  conceptual  knowledge  and 
associated  tasks  for  conducting  a  COIN  operation.  Taking 
approximately  one  to  two  hours  to  complete  out  of  the 
classroom,  the  Primer  prepares  the  student  for  when  they 
enter  the  complex  Practice  Environment.  It  is  broken  into 
eight  lessons,  each  of  which  contain  a  narrative,  interview 
segments  from  former  Commanders,  and  an  actual 
interaction  with  the  Practice  Environment  software  as  a 
means  of  demonstrating  the  task  to  the  student.  The  eight 
lessons  in  the  UrbanSim  Primer  are: 

1.  Population  is  the  center  of  gravity,  how  a 
Commander  can  provide  for  the  population’s 
protection,  welfare,  and  support 

2.  Higher  headquarters’  products :  products  provided 
from  above  that  guide  the  planning  of  the  operation, 
i.e.  Operations  Order,  Commander’s  Critical 
Information  Requirements  (CCIRs),  and  LoEs 

3.  Intelligence  gathering  and  analysis:  running  staff 
estimates  that  allow  a  student  to  conduct  Intelligence 
Preparation  of  the  Battlefield,  including  Political, 
Military,  Economic,  Social,  Information, 
Infrastructure  (PMESII)  and  Sewer,  Water, 
Electricity,  Academics,  Trash,  Transportation, 
Medical,  Fuel  (SWEAT-MF)  analyses 

4.  Navigating  human  terrain:  additional  staff  estimates 
that  includes  human  intelligence  and  social  networks 

5.  Student  products:  student-created  products.  In  the 
case  of  battalion  commanders:  Commander’s  Intent, 
LoEs,  CCIRs,  MOEs,  and  a  running  sync  matrix 

6.  Anticipating  2nd  and  3rd  order  effects:  how  to  use 
the  products  above  to  learn  how  to  anticipate 
intended  and  unintended  effects  in  a  COIN  fight 

7.  Execution:  how  to  employ  the  appropriate  strategy  to 
in  a  COIN  fight  (clear-hold-build,  active  versus 
reactive,  combined  action,  limited  support) 

8.  Assessment:  how  to  measure  success  in  an  ill-defined 
domain  such  as  COIN.  Using  quantitative  and 
qualitative  metrics  to  interpret  LoE  advancement 

One  of  the  unique  features  of  the  UrbanSim  Primer  is 
its  arrangement  around  the  Guided  Experiential  Learning 
model  of  lesson  structure  (Clark,  2004).  For  each  lesson 
in  the  UrbanSim  Primer,  the  student  is  walked  through  a 
series  of  lesson  parts  that  are  derived  from  the 
pedagogical  lesson  format  of  this  model: 

First,  the  student  is  presented  with  the  lesson’s 
learning  objective,  describing  what  the  student  will  be 
able  to  accomplish  in  a  COIN  operation  after  completing 
the  lesson  that  they  were  unable  to  accomplish  before  the 


lesson.  For  example,  the  student  will  learn  how  to  develop 
battalion  commander  products,  such  as  the  Commander’s 
Intent,  CCIRs,  LoEs,  and  MOEs  for  a  stability-focused 
mission.  Second,  the  student  is  presented  with  the  reason 
why  achieving  the  learning  objective  will  enable  the 
Commander’s  success  in  a  COIN  operation.  For  example, 
correctly  employing  LoEs  will  assist  the  Commander  in 
formulating  a  complete  situational  understanding  of  the 
battlefield. 

Third,  the  student  is  presented  with  an  overview,  using 
visual  models  and  a  brief  description,  of  the  lesson’s 
placement  in  the  overall  Primer,  as  well  as  lesson  parts 
that  comprise  it. 

Fourth,  the  student  is  presented  with  the  concepts, 
processes,  and  principles  that  are  the  prerequisite 
knowledge  for  learning  the  specific  procedure  outlined  in 
the  lesson.  For  example,  a  CCIR  would  be  described 
along  with  how  it  is  used  in  a  COIN  setting. 

Fifth,  the  specific  task  or  capability  that  a  Commander 
must  possess  is  demonstrated  or  articulated  through 
interviews,  visual  cues,  and  interactions  with  the  Practice 
Environment  software.  As  an  example  from  the  UrbanSim 
Primer,  a  student  is  presented  with  a  snippet  from  an 
interview  with  a  former  battalion  commander  of  how 
MOEs  allowed  him  to  gauge  success  in  his  area  of 
operation.  The  actual  procedure  for  authoring  an  MOE  in 
the  Practice  Environment  is  then  demonstrated. 

Sixth,  the  student  is  then  directed  to  the  Practice 
Environment  software  where  they  practice  the  procedure 
themselves.  The  environment  in  which  the  practice  takes 
place  begins  with  easier  scenarios,  and  increasingly 
becomes  difficult  as  the  student  completes  various 
lessons.  The  increase  in  complexity  is  cumulative,  with 
the  more  difficult  exercises  building  upon 
knowledge/results  from  the  simpler  exercises. 
Additionally,  guidance  from  the  Primer  weans  as  the 
student  progresses  through  each  part  of  the  lesson.  For 
example,  after  the  player  has  authored  a  Commander’s 
Intent  in  one  lesson  part,  they  will  then  use  that 
Commander’s  Intent  to  identify  their  CCIRs  in  a 
subsequent  lesson  part.  Areas  of  and  interactions  with  the 
Practice  Environment  are  unlocked  as  they  progress 
through  the  Primer  lessons  as  not  to  overwhelm  the 
student  with  the  complexity  of  the  software. 

Seventh,  the  student’s  performance  in  these 
increasingly  difficult  exercises  is  evaluated  to  determine 
if  a)  what  they  did  was  correct,  and  b)  how  they  must 
adjust  their  approach  so  the  learning  goals  are 
accomplished.  In  the  UrbanSim  Primer,  an  automated 
tutor  provides  feedback  on  the  actions  undertaken  by  the 
player,  and  shows  how  a  different  set  of  actions  may  have 
led  to  a  more  desirable  outcome. 


3.2  The  UrbanSim  Practice  Environment 

The  UrbanSim  Practice  Environment  is  a  game-based 
social  simulation  that  allows  a  student  to  plan,  prepare, 
and  execute  a  COIN  operation.  Similar  to  a  turn-based 
strategy  game  (such  as  Civilization  or  Age  of  Empires), 
the  student  directs  subordinate  units  to  take  action  in  a 
virtual  environment,  and  attempts  to  successfully 
complete  a  COIN  operation  using  the  products/strategies 
learned  in  the  Primer. 

The  layout  of  the  Practice  Environment  is  derived 
from  an  amalgamation  of  several  sources  including  CTA 
results,  discussions  with  subject  matter  experts,  U.S. 
Army  doctrine,  and  experiences  directly  from  theatre,  of 
which  the  latter  proved  to  be  particularly  useful.  Though 
the  CTA  identifies  specific  tasks  that  the  student  must 
execute  (conduct  mission  analysis,  deliberate 
reconnaissance,  etc),  it  does  not  always  achieve  the 
precision  necessary  for  implementation  in  the  practice 
environment.  Experiences  from  theatre  provided  many  of 
underspecified  pieces,  such  as  the  use  of  PMESII  and 
SWEAT  diagrams  as  a  means  of  analyzing  the  area  of 
operation.  The  specific  learning  objectives  targeted  for  the 
Practice  Environment  include  how  to  successfully  plan 
for,  prepare  for,  and  execute  a  COIN  operation,  as  well  as 
self-assessing  performance  in  this  complex  domain.  To 
meet  these  learning  objectives,  the  Practice  Environment 
is  divided  into  a  series  of  different  stages  that  the  student 
navigates  through. 

Read  Ahead:  Before  entering  the  actual  Practice 
Environment  application,  the  student  is  provided  with  a 
set  of  read-ahead  material  that  is  used  to  familiarize  them 
with  the  area  of  operations.  This  material  takes  the  form 
of  hard  copy  documentation,  and  is  intended  to  mimic  the 
“left  seat,  right  seat”  exercise  that  Commanders 
participate  in  when  first  deployed  to  theatre  where  the 
departing  Commander  provides  the  incoming  Commander 
with  a  full  overview  and  assessment  of  the  area  of 
operations.  Area  maps,  historical  information,  dossiers, 
target  folders,  and  preliminary  staff  estimates  are  some  of 
the  products  provided  to  the  student  up  to  a  week  before 
playing  through  an  UrbanSim  scenario  as  a  means  of 
immersing  themselves  in  the  physical  and  cultural 
landscape  that  the  virtual  environment  will  simulate.  The 
premise  behind  providing  this  material  ahead  of  time  is  to 
prevent  wasting  valuable  classroom  time  having  a  student 
become  familiar  with  a  complex  area  of  operation. 

Mission  Overview.  After  preparing  with  the  read- 
ahead  material  up  to  a  week  in  advance,  the  student  enters 
the  classroom  and  launches  the  Practice  Environment 
application.  The  scenario  is  loaded  and  the  student  is 
provided  with  the  products  produced  by  higher 
headquarters  that  outline  the  overall  mission  strategies 
and  goals.  This  includes  the  higher  headquarters 


Operations  Order,  Commander’s  Intent,  CCIRs,  and 
LoEs.  Each  of  these  products  provides  a  foundation  for 
the  student  to  develop  their  situational  understanding  of 
the  battlefield,  which  is  then  supplemented  with  the 
products  in  Mission  Analysis. 

Mission  Analysis:  After  familiarizing  themselves  with 
the  higher  headerquarters  products,  students  then  review 
the  information  provided  by  their  immediate  staff. 
Identified  as  running  staff  estimates,  these  products  are 
often  produced  by  Intelligence  (S-2)  and  Operations  (S-3) 
Officers  to  help  the  Commander  formulate  his 
visualization  of  the  area  of  operation.  For  UrbanSim, 
these  products  are  provided  to  assist  the  student  develop 
their  visualization  of  the  entire  spectrum  of  the  mission 
(offense,  defense,  stability),  with  an  emphasis  on  the 
stability  component.  These  products  include  PMESII  and 
SWEAT  analyses,  target  folders  for  key  individuals  and 
groups  in  the  area  of  operation,  and  assorted  network 
overlays.  The  network  overlays  were  initially  designed  as 
a  single  social  network  that  illustrated  the  relationships 
that  exist  between  various  simulation  entities  and 
coalition  forces,  but  it  was  determined  after  discussions 
with  subject  matter  experts  that  several  other  network 
types  were  useful  to  Commanders  in  theatre.  As  a  result, 
additional  networks  were  added  (military,  political,  and 
economic)  that  represent  relationships  not  just  between 
individuals,  but  also  between  groups/organizations,  and 
even  structures.  All  of  the  information  that  comprises 
these  products  are  dynamic,  and  subject  to  updates  as  the 
student  plays  through  an  UrbanSim  scenario  (e.g., 
PMESII-PT  and  SWEAT  analysis  values  that  are  updated 
as  conditions  in  the  area  of  operation  change  over  time). 

Mission  Plan :  After  formulating  their  understanding 
and  visualization  of  the  scenario  area,  the  student  then 
describes  their  understanding  through  the  development  of 
assorted  Commander  products,  which  will  be  used  to 
inform  the  course  of  action  they  plan  to  take.  This 
includes  the  traditional  Commander’s  Intent  and  CCIRs, 
but  also  includes  LoEs  and  MOEs,  which  are  increasingly 
being  employed  by  Commander’s  to  rapidly  update  their 
situational  understanding  as  the  operation  is  executed.. 
One  of  the  central  mechanisms  for  student  assessment  is 
through  LoE  monitoring,  therefore  it  was  required  that  we 
develop  an  approach  that  allowed  LoE  values  to  be 
computationally  determined.  To  accomplish  this,  every 
action  taken  by  both  the  player  and  all  Non-Player 
Characters  (NPCs)  in  a  scenario  has  a  set  of  world  state 
effects  that  in  turn  affect  a  set  of  LoEs  (Civil  Security, 
Governance,  Economic  Development,  Information 
Operations,  Infrastructure,  and  Essential  Services). 
Conflicting  effects  as  a  result  of  student/agent  actions  are 
resolved  through  a  simple  additive  model.  In  this  Mission 
Plan  phase,  the  student  identifies  relevant  LoEs  for  the 
particular  mission,  and  during  runtime  these  LoE  values 
are  dynamically  updated  to  assist  the  student  understand 


their  progress  towards  achieving  the  desired  end  state.  In 
sum,  LoEs  both  provide  an  overall  summary  of  the 
student’s  progress  in  a  mission  (through  the  LoE  values) 
and  play  a  key  role  in  supporting  decision-making  (the 
intelligent  tutor’s  use  of  LoE’s  is  discussed  in  the  next 
section). 

Directing  Action:  After  developing  their 

Commander’s  products,  the  student  then  enters  the  direct 
phase,  which  is  analogous  to  a  turn-based  strategy  game. 
The  student  directs  subordinate  units  to  take  action  on  key 
individuals,  groups,  and  structures  in  the  scenario  in  an 
attempt  to  reach  their  desired  end  state  as  articulated  in 
the  Commander’s  Intent.  Each  turn  cycle  represents  one- 
day  in  simulation  time,  though  actions  can  take  multiple 
turns  (i.e.,  days),  and  can  be  interrupted  if  conditions  in 
the  world  do  not  allow  the  action  to  complete  (e.g., 
money  runs  out  to  construct  a  school).  The  scenario  ends 
when  one  of  three  conditions  becomes  true:  1)  the 
specified  mission  duration  ends  (i.e.,  run  out  of  turns),  2) 
the  student  (or  instructor)  terminates  the  session 
prematurely,  or  3)  the  performance  of  the  student 
diminishes  significantly  over  a  period  of  time  as  a  result 
of  increased  violence  and  decreased  population  support 
for  the  host  nation. 

The  underlying  agent  system  controlling  the  actions  of 
NPCs  is  PsychSim,  which  directs  action  based  on 
decision-theoretic  reasoning  and  allows  for  the 
transparent  inspection  of  declarative  models  of  NPCs 
goals  and  beliefs  (Pynadath  &  Maresella,  2005). 
Additionally,  the  agent  models  can  be  inverted  to  generate 
explanations  and  causal  chains  to  reason  about  why 
agents  performed  the  actions  they  did.  This  feature  is 
critical  to  providing  intelligent  guidance  for  the  student, 
which  is  described  in  Section  4.  Furthermore,  the  agents 
are  capable  of  reasoning  about  the  responses  that  other 
agents  will  have  to  their  actions,  which  significantly 
increases  the  complexity  of  the  state  space  but  also 
affords  a  level  of  reasoning  not  seen  with  many 
simulation  systems  today. 

Mission  Debrief:  The  final  phase  of  the  Practice 
Environment  is  the  Mission  Debrief,  which  summarizes 
and  analyzes  the  student’s  progress  in  the  scenario.  This 
includes  bar  charts  and  graphs  illustrating  the  student’s 
progress  as  it  relates  to  LoEs  and  MOEs  over  time,  as 
well  as  the  amount  of  time  they  spent  in  each  of  the 
various  displays  (spatial,  network,  planning, 
performance).  This  last  assessment  metric  is  intended  to 
inform  the  student  that  they  should  not  be  focusing  their 
attention  on  just  one  or  two  displays,  but  rather  dividing 
their  time  equally  across  all  as  a  means  of  developing  a 
complete  situational  understanding  of  the  area  of 
operation.  The  primary  metric  used  to  inform  the 
student’s  success  in  a  scenario  is  the  value  of  the 
population’s  support  for  the  Host  Nation  government. 


This  is  represented  as  a  Population  Support  Meter,  and  is 
available  to  the  student  throughout  the  scenario,  but  is 
also  used  as  an  important  assessment  tool  at  the 
completion  of  a  scenario.  This  value  is  updated  similar  to 
the  way  the  LoE  values  are  updated:  every  agent  in  the 
scenario  has  a  “support  for  Host  Nation”  state  feature, 
which  increases  or  decreases  depending  upon  the  actions 
taken  to  affect  it. 

4.  INTELLIGENT  TUTORING 

The  UrbanSim  practice  environment  presents  a  huge 
problem  space,  with  many  solution  paths  of  varying 
degrees  of  quality.  This  complexity  is  a  strength  in  that  it 
attempts  to  provide  a  realistic  practice  environment 
through  modeling  of  human  behavior,  but  also  presents 
challenges  in  terms  of  learning.  Specifically,  large  open 
learning  environments  that  rely  on  discovery  learning  can 
be  problematic  for  early-stage  learners  (Kirschner  et  al., 
2006).  The  need  for  guidance  is  a  reoccurring  and 
established  principle  of  instructional  design  (Merrill, 
2002)  and  is  delivered  in  two  key  forms:  through  the 
Primer  (discussed  in  section  3)  and  by  an  intelligent 
tutoring  system  (ITS)  that  provides  feedback  in  the 
practice  environment. 

Expert  human  tutors  and  the  best  ITSs  deliver 
formative  feedback  -  that  is,  “information  communicated 
to  the  learner  that  is  intended  to  modify  his  or  her 
thinking  or  behavior  to  improve  learning”  (Shute,  2008). 
Explicit  feedback  can  be  used  for  a  variety  of  reasons, 
such  as  to  verify  the  correctness  of  an  action,  explain 
correct  answers,  remediate  misconceptions,  reveal  goal 
structure,  and  more.  Feedback  can  be  delivered 
immediately  after  an  action,  or  after  some  delay.  The  best 
choices  for  feedback  content  and  timing  depend  on  many 
things,  including  task  domain,  nature  of  the  skill  being 
learned,  the  aptitude  of  the  learner,  whether  the  learner 
has  a  performance  orientation,  and  more  (Shute,  2008). 
The  UrbanSim  ITS  is  designed  to  support  both  immediate 
and  delayed  feedback. 

As  discussed,  one  of  the  goals  of  UrbanSim  is  to  teach 
about  the  broader  and  unintended  effects  of  actions  taken 
in  stability-focused  operations  (fourth  learning  objective: 
directing  action  to  anticipate  2nd  and  3rd  order  effects). 
Understanding  the  role  of  NPCs  in  the  PsychSim  models 
is  part  of  this.  Each  NPC  agent  acts  to  achieve  its  goals 
and  makes  decisions  based  on  the  state  of  the  world. 
Although  the  student  cannot  directly  order  non-U. S. 
NPCs  to  take  (or  not  take)  certain  actions,  she  or  he  can 
certainly  affect  the  world  state.  A  key  goal  for  the  ITS  is 
to  help  the  learner  understand  this  idea,  and  to  take 
actions  that  1)  limit  the  ability  of  NPCs  to  take  harmful 
actions,  and  2)  enable  NPCs  to  take  helpful  actions.  In 
other  words,  the  learner  should  be  thinking  about  how 
their  actions  influence  the  actions  of  others  -  the  ITS 


frames  its  feedback  in  this  light  and  attempts  to  reveal  the 
reasoning  behind  NPC’s  actions:  why  they  made  the 
decisions  they  made,  what  consequences  (seen  or  unseen) 
were  most  relevant,  and  under  what  circumstances 
different  decisions  would  have  been  made. 

To  support  learning  of  unintended  consequences,  we 
have  implemented  an  anticipate-wait-relate  tutoring 
strategy.  That  is,  after  the  learner  has  proposed  an  action 
and  the  ITS  has  decided  to  apply  the  strategy,  three  steps 
are  taken:  (1)  Elicit  the  anticipated  effects  of  that  action. 
That  is,  ask  the  learner  to  assess  that  choice  by  indicating, 
via  drop  down  menus,  how  she  or  he  expects  that  action 
to  affect  the  world  state  or  relevant  LoEs.  (2)  After  this 
input,  allow  the  game  to  proceed  for  some  number  of 
turns  (which  is  only  1  at  the  time  of  this  writing,  but 
longer  delays  are  possible).  (3)  Finally,  the  ITS  presents 
the  learner  with  the  actual  results  for  comparison. 

The  system  is  able  to  provide  feedback  before  or  after 
the  initial  action  proposal,  or  later  (step  3),  along  with  the 
comparison  between  anticipated  and  actual  outcomes.  Our 
focus  thus  far  has  been  on  this  delayed  form  of  feedback. 
Application  of  this  strategy  requires  answers  to  at  least 
two  questions.  First,  what  is  used  to  trigger  the  strategy? 
In  other  words,  when  should  the  learner  be  prompted  to 
anticipate  the  effects  of  an  action?  Second,  how  should 
the  ITS  support  reflection  on  the  results  of  the 
comparison?  Of  course,  the  learner  could  easily  be  asked 
to  anticipate  outcomes  to  every  action,  but  this  would 
quickly  become  a  distraction.  Also,  learning  could 
potentially  occur  by  simply  allowing  the  learner  to  inspect 
the  predicted  versus  actual  outcomes  and  learn  from  them. 
This  also  is  unappealing,  especially  given  the  rich 
PsychSim  models  that  drive  NPC  behavior.  In  fact,  our 
approach  leverages  these  models  and  the  reasoning 
capabilities  of  PsychSim. 

Our  initial  approach  to  answering  the  question  of 
when  to  ask  the  learner  to  anticipate  is  to  use  look-ahead. 
If  an  LoE  is  about  to  decrease  from  a  user  action,  the  ITS 
will  ask  the  learner  to  anticipate  either  the  effect  on  the 
LoE  or  world  state  features  that  contribute  to  the  LoE 
calculation.  We  currently  focus  on  potentially  damaging 
actions  the  learner  can  take.  Regarding  feedback,  we  have 
implemented  an  approach  based  on  causal  chains  of  the 
reasoning  behind  the  NPCs.  These  causal  chains  reveal 
the  state  changes  that  occur  based  on  learner  actions, 
allowing  the  learner  to  see  the  connections  between  their 
actions,  the  world  state,  and  the  ensuing  NPC  actions. 
Additionally,  the  ITS  also  queries  PsychSim  to  reveal 
what  conditions  would  have  led  to  different  NPC  actions. 
For  example,  an  NPC’s  ability  to  have  taken  one  very  bad 
action  may  have  been  impossible  had  the  learner  taken  a 
different  action  at  that  turn.  The  ITS’s  aim  here  is  to 
support  the  learner’s  reflection  in  imagining  what  other 
actions  could  have  produced  such  world  states. 


5.  ASSESSMENT 

Assessing  student  performance  in  this  instructional 
package  occurs  through  several  forms.  First,  students 
judge  their  own  progress  during  missions  by  studying 
MOEs  and  MOPs  and  relating  them  to  identified  LoEs. 
Second,  population  modeling  provides  the  lower-level 
information  necessary  for  automatically  measuring  the 
student's  performance  within  a  scenario.  Third,  intelligent 
assessment  of  learning  is  supported  by  tracking  student 
recognition  of  important  events,  including  the  anticipation 
of  2nd  and  3rd  order  effects  that  the  ITS  supports. 

In  future  work,  we  intend  to  determine  empirically  the 
relation  between  patterns  of  decision-making  and 
outcome  measures  (were  the  LOEs  advanced  or  not?).  If 
particular  patterns  lead  consistently  to  positive  outcomes, 
whereas  others  lead  to  negative  outcomes,  and  we  are  able 
to  automatically  monitor  each  student's  pattern, that  would 
provide  a  basis  on  which  to  provide  each  student 
customized  feedback  intended  to  shift  their  pattern  closer 
to  one  leading  to  positive  outcomes.  In  his  book  Logic  of 
Failure,  Dorner  (1996)  summarized  research  on  policy 
making  in  the  context  of  complex  socio-economic 
simulations.  He  categorized  his  participants'  "think-out- 
load"  protocols  as  reflecting  different  types  of  cognitive 
processing.  He  found  that  the  relative  proportion  of  time 
spent  using  the  different  processes  followed  a  different 
pattern  in  "good"  participants  as  opposed  to  "bad" 
participants,  where  good  and  bad  related  to  the 
participants'  ability  to  make  the  simulated  society  prosper 
as  opposed  to  collapse.  Analogously,  we  intend  to  collect 
think-out-load  protocols  from  Soldiers  interacting  with 
Urbansim,  identify  from  these  the  different  cognitive 
processes  engaged  (e.g.,  information  seeking  or  decision 
making),  and  examine  how  the  relative  time  spent  in  each 
process  changes  over  the  course  of  working  the  scenario. 
If  there  is  a  link  between  certain  patterns  and  positive  vs. 
negative  outcomes,  we  would  then  need  to  determine  how 
to  refine  the  user-interface  so  as  to  capture  these  patterns. 
Additional  think-out-loud  protocols  would  then  be 
required  to  verify  the  relation  between  the  actual  thought 
pattern  and  the  pattern  inferred  from  user-interaction  data. 

Analysis  of  think-out-load  protocols  has  an  additional 
benefit  by  providing  a  means  of  validating  the  training 
system  itself.  Players  who  demonstrate  patterns  of 
thinking  proscribed  by  the  learning  objectives  ought  to  be 
successful  in  moving  the  world  state  toward  a  desired  end 
state.  If  not,  then  the  instructional  and  system  designs 
must  be  refined  as  discussed  in  section  2.8. 
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